<!--
To change this template, choose Tools | Templates
and open the template in the editor.
-->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>
    <title></title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
  
<b>JObjectTree</b>: A very simple scheme for storing Java objects in file systems and S3.
<p>
<i>JObjectTree</i> is a very simple scheme for storing Java objects. Objects are organized
in filesystem-like hierarchical trees.  Any
<a href="http://java.sun.com/javase/6/docs/api/java/io/Serializable.html">Serializable</a>
object can be stored in <i>JObjectTree</i> without modification.  <i>JObjectTree</i> is designed to use
multiple underlying object stores and currently supports two: a local file system and (more importantly),
<a href="http://aws.amazon.com/s3/">Amazon Simple Storage Service (Amazon S3)</a>
JObjectTree  is implemented as a Java library.
<p>
In <i>JObjectTree</i>, each object is uniquely identified by a
<tt><a href="ot/Loc.html">Loc</a></tt>,
which consists of a Context (such as an S3 bucket) and a Path, where a Path is an ordered array of strings.
<p>
By default, <i>JObjectTree</i> uses the standard Java serialization mechanism to convert objects to
a persistable form.  You can control this process by implementing your own version of
<tt><a href="ot/Serializer.html">Serializer</a></tt>.
Currently, the distribution contains two Serializers, one that does plain serialization and another
that compresses objects (see
<tt><a href="ot/GzipSerializer.html">GzipSerializer</a></tt>.
).

<hr style="width: 100%; height: 2px;"><br>
    
<b>Why <i>another</i> Java object persistence scheme?</b>
<p>
There are, at this point, many sophisticated, powerful systems for persisting Java objects.
Despite this, I decided to write <i>JObjectTree</i>, and did so for two main reasons.
First, for projects that entail parallel processing, I find it convenient to store objects in
<a href="http://aws.amazon.com/s3/">AWS S3</a>
(which inherently supports large scale parallel access to data).  Persistence schemes such as
<a href="http://db.apache.org/jdo/jdo_v_jpa.html">JDO</a>
and
<a href="http://java.sun.com/developer/technicalArticles/J2EE/jpa/">JPA</a>
primarily use relational databases as object stores, so they cannot be used with S3.
Secondly, these powerful systems are simply overkill for many of my projects.  I often write
applications that don't need to do queries or ACID transactions - they just need to store objects
in an orderly way and be able to retrieve them reliably.  Since I don't need the flexibility these
systems provide, I really don't want to bother adding annotation to my code or writing XML metadata
files. Hence <i>JObjectTree</i>.
<hr style="width: 100%; height: 2px;"><br>
<b>Dependencies</b>
<p>
JObjectTree uses the following Java libraries, which are included in the distribution zip file
(in the lib directory).
<ul>
<li>
<a href="http://jets3t.s3.amazonaws.com/toolkit/toolkit.html">JetS3t</a>
</li>
<li>
<a href="http://hc.apache.org/httpclient-3.x/">Jakarta Commons HttpClient</a>
</li>
<li><a href="http://commons.apache.org/codec/">Apache Commons Codec</a>
</li>
<li><a href="http://commons.apache.org/logging/">Apache CommonsLogging</a></li>
</ul>

<hr style="width: 100%; height: 2px;"><br>

<b>How to use JObjTree</b>
<p>
This is quick introduction to using <i>JObjectTree</i> with
<a href="http://aws.amazon.com/s3/">AWS S3</a>
as the object store.  For more details, see the Javadoc API documentation.
<p>
<u>Get the jars</u>
<p>
All the jars you need are in <tt>lib</tt> directory of the zip distribution file.  Put them in your classpath.
<p>
<u>Get access to S3</u>
<p>
To use <i>JObjectTree</i> with S3, you'll need to specify your AWS access ID and AWS secret key.
Once you have an
<a href="http://aws.amazon.com/account/">AWS account</a>,
you can find these on your
<i>Access Identifiers page</i>.
<p>
<u>Create an S3 bucket</u>
<p>
All operations are performed using an instance of instance of
<tt><a href="ot/Ot.html">Ot</a></tt>.
An <tt>Ot</tt> instance handles access to one 
<tt><a href="ot/Context.html">Context</a></tt>.
For an S3 context (
<tt><a href="ot/s3/ContextS3.html">ContextS3</a></tt>.
), this means one S3 bucket.
<p>
It's best to have a bucket that's dedicated to <i>JObjectTree</i>.
While this isn't absolutely necessary, but it makes things much simpler, and there's no
cost to creating a bucket.  You can create a bucket programmatically (see
<tt><a href="ot/s3/OtS3.html#createBucket(java.lang.String)">OtS3.createBucket()</a></tt>
) but it's just as easy to create one using an interactive tool, such as
<a href="https://addons.mozilla.org/en-US/firefox/addon/3247">S3 Organizer</a>.
See the 
<a href="http://docs.amazonwebservices.com/AmazonS3/2006-03-01/">AWS S3 API Reference Guide</a>
for bucketname rules (e.g. underscores are not allowed).
<p>
<u>Create an OtS3 instance</u>
<blockquote><tt>
    Ot ot = new OtS3(new ContextS3("your-bucket-name"),"aws_access_ID", "aws_secret_key");
</tt></blockquote>
Or, to compress your objects:
<blockquote><tt>
    Ot ot = new OtS3(new ContextS3("your-bucket-name"),"aws_access_ID", "aws_secret_key", new GZipSerializer());
</tt></blockquote>
<p>
<u>Store objects</u>
<p>
To store an object, you need to specify an
<tt><a href="ot/OPath.html">OPath</a></tt>,
which is just an ordered array of strings.  The last component of the path should have an
appropriate extension, e.g.
<blockquote><tt>
    String objectNameExtension = ot.getContext().getDefaultSerializer().getObjectNameExtension();
</tt></blockquote>
Then
<blockquote><tt>
    OPath path = new OPath(new String[] { "path", "to","an_object." +  objectNameExtension });
</tt></blockquote>
And to store an object with this path:
<blockquote><tt>
    Serializable x = new SomeSerializableClass();<br>
    ot.add(path,a);
</tt></blockquote>
(If you have classes that will often be stored in <i>JObjectTree</i>, it's convenient to have
them create their own paths by implementing the
<tt><a href="ot/Otable.html">Otable</a></tt>
interface.)
<p>
<u>List objects</u>
<p>
To list all the objects in a context,
<blockquote><tt>
    java.util.List<Loc> objLocs = ot.listContext();
</tt></blockquote>
See
<tt><a href="ot/Ot.html">Ot</a></tt>
for other listing methods.
<p>
<u>Retrieve objects</u>
<p>
<blockquote><tt>
    Serializable obj = ot.get(path);
</tt></blockquote>
<p>
<u>Other operations</u>
<p>
See
<tt><a href="ot/Ot.html">Ot</a></tt>
for other operations, such as testing for the existence of an object and deleting an object.
See
<tt><a href="ot/s3/OtS3.html">OtS3</a></tt>
for a couple of S3 specific operations. For synchronizing an S3 context with a file system context,
see 
<tt><a href="ot/Sync.html">ot.Sync</a></tt>.

<hr style="width: 100%; height: 2px;"><br>

<b>Security</b>
<p>
<blockquote>
<b>SSH</b>
<br>
By default, <tt>OtS3</tt> communicates with S3 en clair. To enable SSH, you need to reset a
<a href="http://jets3t.s3.amazonaws.com/toolkit/toolkit.html">JetS3t</a>
property (JetS3T is the Java library used to communicate with S3).  You can reset
JetS3t properties with OtS3.setJetS3tProperty().  Use
<blockquote><tt>
    ot.setJet3tProperty("s3service.https-only","true");
</tt></blockquote>
to enable SSH.
<p>
<b>Implementing encryption</b>
<br>
<i>JObjectTree</i> does not yet support encryption.  You can implement your own encryption by
creating a class that implements the
<tt><a href="ot/Serializer.html">Serializer</a></tt>
interface and performs encryption.
In <i>JObjectTree</i>, objects are  written to S3 or files as byte arrays, and these byte arrays
are created by the
<tt><a href="ot/Serializer.html#objectToByteArray(java.io.Serializable obj)">Serializer.objectToByteArray()</a></tt>
method.
When objects are retrieved, they are reconverted to objects by the
<tt><a href="ot/Serializer.html#byteArrayToObject(byte[] B)">Serializer.byteArrayToObject()</a></tt>
method. For examples of how this is done, see the source code for
<tt><a href="ot/StandardSerializer.html">StandardSerializer</a></tt>
and
<tt><a href="ot/GzipSerializer.html">GzipSerializer</a></tt>.
</blockquote>

<hr style="width: 100%; height: 2px;"><br>
    
<b>Caveat re S3</b>
<p>
If you haven't used S3 before, there is a possible pitfall re locking, or rather, the lack thereof.
S3 does not provide any means for client applications to lock an object. Furthermore, S3 makes
extensive use of caches.  This means that if you modify an object at time t0, a read at time
t0 + delta could still get the old version.  There is no way to determine when a write has been
completed. And there is no way to determine how long it will take for a change to be propagated
to caches.
<p>
Internally, S3 does prevent simultaneous writes - once a write command is received, the object's
location is locked until the write is completed.  Writes to a given address are fully processed
in the order received.  We do not have access to this locking mechanism.

<hr style="width: 100%; height: 2px;"><br>

<b>serialVersionUID</b>
<p>
<i>JObjectTree</i> uses the standard Java serialization mechanism and thus is subject to the
standard <tt>serialVersionUID</tt> problem.  For those who haven't used serialization:
each Java class has a version number, the <tt>serialVersionUID</tt>.  By default, this number is
changed (by the compiler) whenever a class is recompiled. This means that if you serialize and
object and then recompile its source, you won't be able to deserialize it because the serialized
object's ID no longer matches the class' ID.  The standard solution it to explicitly set the
<tt>serialVersionUID</tt> of a Serializable class, e.g.
<tt><blockquote>
    static final long serialVersionUID = 1;
</blockquote></tt>

<hr style="width: 100%; height: 2px;"><br>

<b>Logging</b>
<p>
<i>JObjectTree</i>itself does not perform any logging.  However,
<a href="http://jets3t.s3.amazonaws.com/toolkit/toolkit.html">JetS3t</a>,
which is used for communications with S3, uses
<a href="http://commons.apache.org/logging/">Apache Commons Logging</a>.
By default, all log messages are written to the stderr.  To completely disable log messages,
you can use
<tt><a href="ot/s3/NilLog.html">ot.s3.NilLog</a></tt>. Add the line
<tt><blockquote>
    System.setProperty("org.apache.commons.logging.Log", "ot.s3.NilLog");
</blockquote></tt>
at the beginning of your application.  If you want to record log messages, you'll need to add
a logging package that implements Apache Commons Logging.
<a href="http://logging.apache.org/log4j/1.2/index.html">Log4J</a>
is the most common choice.

<hr style="width: 100%; height: 2px;"><br>

<b>Acknowledgement</b>
<p>
For all communications with S3, I rely on James Murty's admirable toolkit,
<a href="http://jets3t.s3.amazonaws.com/toolkit/toolkit.html">JetS3t</a>.

  </body>
</html>
